Research on Robust Audio-Visual Speech Recognition Algorithms

Automatic speech recognition (ASR) that relies on audio input suffers from significant degradation in noisy conditions and is particularly vulnerable to interference. However, video recordings of capture both visual signals, providing a potent source information for training models. Audiovisual (AVSR) systems enhance the robustness ASR by incorporating lip movements associated sound production addition auditory input. There are many audiovisual models transcription, but most them have been tested based single experimental setting with limited dataset. good model should be applicable any scenario. Our main contributions are: (i) Reproducing three best-performing current AVSR research area using famous databases, LSR2 (Lip Reading Sentences 2) LSR3 3), comparing analyzing their performances under various noise conditions. (ii) Based our experiences, we analyzed problems currently encountered domain, which summarized as feature-extraction problem domain-generalization problem. (iii) According results, Moco (momentum contrast) + word2vec (word vector) has best effect LRS datasets regardless whether there or not. Additionally, also produced results experiments recognition. lays foundation further improving performance